Papers with large batch training
Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Recent research shows that fine-tuning dense retrievers to realize their capacity requires carefully designed fine-cuning techniques. |
| Approach: | They propose a pre-training architecture that learns to condense information into the dense vector through LM pre-training and a coCondenser architecture which adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. |
| Outcome: | The proposed architecture reduces the need for heavy data engineering and large batch training. |